Embodiments of the inventive subject matter generally relate to the field of data processing, or, more specifically, methods, apparatus, and products for reliable messaging using redundant message streams in a high speed, low latency data communications environment.
Messaging environments are generally available to provide data communication between message sending devices and message receiving devices using application messages. An application message is a quantity of data that includes one or more data fields and is passed from a message producer installed on a message sending device to a message consumer installed on a message receiving device. An application message is a form of message recognized by application software operating in the application layer of a data communication protocol stack—as contrasted for example with a transport message or network message, which are forms of messages recognized in the transport layer and the network layer respectively. An application message may represent, for example, numeric or textual information, images, encrypted information, and computer program instructions.
A messaging environment may support point-to-point messaging, publish and subscribe messaging, or both. In a point-to-point messaging environment, a message producer may address an application message to a single message consumer. In a publish and subscribe messaging environment, a message producer may publish an application message to a particular channel or topic and any message consumer that subscribes to that channel or topic receives the message. Because message producers and message consumers communicate indirectly with each other via a channel or topic in a publish and subscribe environment, message transmission is decoupled from message reception. As a consequence, neither producers nor consumers need to maintain state about each other, and dependencies between the interacting participants are reduced or eliminated. A publish and subscribe environment may, therefore, allow message publishers and message subscribers to operate asynchronously.
For further explanation of a messaging environment, FIG. 1 sets forth a block diagram illustrating a typical messaging environment for data communications that includes a message sending device (100), a message receiving device (104), and a message administration server (102). The message sending device (100) is a computer device having installed upon it a message producer (110), a set of computer program instructions configured for transmitting application messages to the message administration server (102) for delivery to a message receiving device. In the example of FIG. 1, the message producer (110) transmits application messages to the message administration server (102) on a message stream (106). The message sending device (100) may produce the transmitted messages by generating the application messages from data of the message sending device itself or data received from some other source. The message receiving device (104) is a computer device having installed upon it a message consumer (112), a set of computer program instructions configured for receiving application messages from the message administration server (102). In the example of FIG. 1, the message consumer (112) receives the application messages from the message administration server (102) on a message stream (108). In the example of FIG. 1, the message stream (106) and the message stream (108) are data communication channels implemented using, for example, the User Datagram Protocol (‘UDP’) and the Internet Protocol (‘IP’).
In either a point-to-point messaging environment or a publish and subscribe messaging environment, the application messages transmitted from message sending devices to message receiving devices typically pass through the message administration server (102). The message administration server (102) is computer device having installed upon it a message administration module (114), computer program instructions configured for administering the messages transmitted from the message producer (110) to the message consumer (112). Examples of message administration modules may include the IBM WebSphere® MQ, the Open Message Queue from Sun Microsystems, and the OpenJMS from The OpenJMS Group. In a point-to-point messaging environment, the message administration module (114) provides message queuing for the message consumer (112) as the message administration module (114) receives application messages addressed to the consumer (112) from various message providers. In a publish and subscribe messaging environment, the message administration module (114) administers the various channels or topics to which message producers publish and message consumers subscribe. In either message environment, the message administration module (114) may also provide security services to ensure that the only messages arriving at the messaging consumer (112) from the message producer (110) are those messages that the message consumer (112) is authorized to receive and that the message producer (110) is authorized to send. Moreover, the message administration module (114) may also coordinate providing to the message consumer backup messages from a backup message producer in the event that a failure occurs on the message producer (110).
Current messaging environments such as, for example, the one described above with reference to FIG. 1, have certain drawbacks. Application messages transmitted to a message administration server from a message sending device for delivery to a message receiving device are delayed in the message administration server until the message administration server can process the messages. The message processing that occurs in the message administration server increases the overall messaging latency of the messaging environment and decreases the overall speed for transmitting data in the data communications environment. Messaging latency is the time period beginning when a message producer transmits an application message and ending when a message consumer receives the application message.
In many data communication environments, even slight increases in messaging latency are costly. Consider, for example, a financial market data environment. A financial market data environment is a data processing environment used to communicate information about financial markets and participants in financial markets. In a financial market data environment, an application message is commonly referred to as a ‘tick’ and represents financial market data such as, for example, financial quotes or financial news. Financial quotes include bid and ask prices for any given financial security. A ‘bid’ refers to the highest price a buyer is willing to pay for a security. An ‘ask’ refers to the lowest price a seller is willing to accept for a security. In a financial market data environment, a message producer may provide quotes for the purchase or sale of financial securities based on real-time financial market conditions, and a message consumer may buy and sell financial securities based on financial quotes. When a message consumer buys or sells a financial security based on the quoted price provided by the message producer, the ability of a message consumer to obtain the bid or ask in the quote for the financial security is largely influenced by messaging latency in the financial market data environment. The higher the messaging latency, the less likely a buy or sell order generated by the message consumer will execute at or near the price stated in the financial quote. In fact, a highly volatile security may fluctuate in price dramatically over a time period of a few seconds.
Current solutions to reduce messaging latency are to remove the message administration server from the messaging environment. In such current solutions, the message sending devices send application messages directly to message receiving devices. The drawback to such current solutions is that removing the message administration server removes the administration functionality provided by the message administration server from the messaging environment. Current solutions, therefore, effectively offer no solution in messaging environments where the administrative functions of a message administration server are required. Consider again the financial market data environment example from above. In such an exemplary financial market data environment, consider that a message receiving device is only authorized to receive financial quotes on certain financial securities or only authorized to receive financial quotes that are at least fifteen minutes old. Removing the message administration server from such a financial market data environment removes the ability to administer the messages received by the message receiving device from the message sending device in the financial market data environment.
An additional drawback to current messaging environments, such as, for example, the one describe above with reference to FIG. 1, involves the situation in which a message administration server receives application messages from both an active message sending device and a backup message sending device, and each message sending device generates the application messages from data received from a plurality of sub-sources. As each message sending device generates the application messages, each message is assigned a message sequence number. Because no guarantee exists that each message sending device will process the data received from the plurality of sub-sources in the same order, the message sending devices often assign the same message sequence number to messages containing different data. When the active message sending device fails, the message administration server receives application messages generated from the backup message sending device and must perform data recovery to identify lost or duplicate data. Because each message sending device may assign the same message sequence number to messages containing different data, message administration servers in current messaging environments cannot rely on the message sequence number to perform data recovery such as eliminating duplicate application messages received from the backup message sending device. Rather, message administration servers must examine the entire contents of each message to determine whether a message administration server has received a duplicate application message during failover. Examining the entire contents of the application message, however, greatly increases messaging latency during a failover from an active message sending device to a backup message sending device. A need, therefore, exists to more efficiently ensure correct and consistent data identification by the message administration server of the contents of the application messages to perform data recovery.